
<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

    <title>Administrator debugging &#8212; LAVA 2024.05 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="_static/doctools.js"></script>
    <script src="_static/sphinx_highlight.js"></script>
    <link rel="shortcut icon" href="_static/favicon.ico"/>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Administering LAVA using Docker" href="docker-admin.html" />
    <link rel="prev" title="Administering VLANd support in LAVA" href="vland-admin.html" />
    <link rel="canonical" href="https://docs.lavasoftware.org/lava/pipeline-debug.html" />
  
<meta charset='utf-8'>
<meta http-equiv='X-UA-Compatible' content='IE=edge,chrome=1'>
<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1'>
<meta name="apple-mobile-web-app-capable" content="yes">
<script type="text/javascript" src="_static/js/jquery-1.12.4.min.js"></script>
<script type="text/javascript" src="_static/js/jquery-fix.js"></script>
<script type="text/javascript" src="_static/bootstrap-3.4.1/js/bootstrap.min.js"></script>
<script type="text/javascript" src="_static/bootstrap-sphinx.js"></script>


  </head><body>

  <div id="navbar" class="navbar navbar-default navbar-fixed-top">
    <div class="container">
      <div class="navbar-header">
        <!-- .btn-navbar is used as the toggle for collapsed navbar content -->
        <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
        </button>
        <a class="navbar-brand" href="index.html"><span><img src="_static/lava.png"></span>
          LAVA</a>
        <span class="navbar-text navbar-version pull-left"><b>2024.05</b></span>
      </div>

        <div class="collapse navbar-collapse nav-collapse">
          <ul class="nav navbar-nav">
            
                <li><a href="genindex.html">Index</a></li>
                <li><a href="contents.html">Contents</a></li>
            
            
              <li class="dropdown globaltoc-container">
  <a role="button"
     id="dLabelGlobalToc"
     data-toggle="dropdown"
     data-target="#"
     href="index.html">Site <b class="caret"></b></a>
  <ul class="dropdown-menu globaltoc"
      role="menu"
      aria-labelledby="dLabelGlobalToc"><ul class="current">
<li class="toctree-l1"><a class="reference internal" href="index.html">Introduction to LAVA</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="contents.html">Contents</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="glossary.html">Glossary of terms</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="support.html">Getting support</a></li>
</ul>
</ul>
</li>
              
                <li class="dropdown">
  <a role="button"
     id="dLabelLocalToc"
     data-toggle="dropdown"
     data-target="#"
     href="#">Page <b class="caret"></b></a>
  <ul class="dropdown-menu localtoc"
      role="menu"
      aria-labelledby="dLabelLocalToc"><ul>
<li><a class="reference internal" href="#">Administrator debugging</a><ul>
<li><a class="reference internal" href="#debugging-a-lava-instance">Debugging a LAVA instance</a><ul>
<li><a class="reference internal" href="#components">Components</a></li>
<li><a class="reference internal" href="#log-files">Log files</a></li>
<li><a class="reference internal" href="#command-line-debugging">Command line debugging</a></li>
<li><a class="reference internal" href="#configuration-files">Configuration files</a></li>
<li><a class="reference internal" href="#check-lava-instance">Check LAVA instance</a></li>
<li><a class="reference internal" href="#displaying-current-settings">Displaying current settings</a><ul>
<li><a class="reference internal" href="#django-support-in-1-11">Django support in 1.11</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</ul>
</li>
              
            
            
              
                
  <li>
    <a href="vland-admin.html" title="Previous Chapter: Administering VLANd support in LAVA"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">&laquo; Administering...</span>
    </a>
  </li>
  <li>
    <a href="docker-admin.html" title="Next Chapter: Administering LAVA using Docker"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Administering... &raquo;</span>
    </a>
  </li>
              
            
            
            
            
              <li class="hidden-sm"></li>
            
          </ul>

          
            
<form class="navbar-form navbar-right" action="search.html" method="get">
 <div class="form-group">
  <input type="text" name="q" class="form-control" placeholder="Search" />
 </div>
  <input type="hidden" name="check_keywords" value="yes" />
  <input type="hidden" name="area" value="default" />
</form>
          
        </div>
    </div>
  </div>

<div class="container">
  <div class="row">
    <div class="body col-md-12 content" role="main">
      
  <section id="administrator-debugging">
<span id="debugging-v2"></span><span id="index-0"></span><h1>Administrator debugging<a class="headerlink" href="#administrator-debugging" title="Permalink to this heading">¶</a></h1>
<section id="debugging-a-lava-instance">
<h2>Debugging a LAVA instance<a class="headerlink" href="#debugging-a-lava-instance" title="Permalink to this heading">¶</a></h2>
<section id="components">
<span id="debugging-components"></span><h3>Components<a class="headerlink" href="#components" title="Permalink to this heading">¶</a></h3>
<p>Each of these components has a service which may need to be restarted when
making changes. Each of these services are restarted when the relevant packages
are installed.</p>
<ul>
<li><p><strong>lava-server</strong> - the frontend UI and admin interface. If using apache
use <code class="docutils literal notranslate"><span class="pre">apache2ctl</span> <span class="pre">restart</span></code> when changing any of the django files, device type
templates or lava-server settings:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo apache2ctl restart
</pre></div>
</div>
</li>
<li><p><strong>worker</strong> - each dispatcher worker connects to the server using HTTP and
follows the instructions of the server, using configuration specified by the
server. Restart is rarely needed, usually only when changing the dispatcher
code related to HTTP or the loghandler:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo service lava-worker restart
</pre></div>
</div>
</li>
</ul>
<p>The worker have dedicated singleton processe which should be put
into loglevel <code class="docutils literal notranslate"><span class="pre">DEBUG</span></code> when investigating problems.
Restart the service after editing the service file.</p>
<ul class="simple">
<li><p><strong>worker</strong> <code class="docutils literal notranslate"><span class="pre">/etc/init.d/lava-worker</span></code> currently defaults to DEBUG.</p></li>
</ul>
</section>
<section id="log-files">
<h3>Log files<a class="headerlink" href="#log-files" title="Permalink to this heading">¶</a></h3>
<p>All log files use <code class="docutils literal notranslate"><span class="pre">logrotate</span></code>, so the information you need may be in a
<code class="docutils literal notranslate"><span class="pre">log.1</span></code> or <code class="docutils literal notranslate"><span class="pre">log.2.gz</span></code> file - typically up to <code class="docutils literal notranslate"><span class="pre">log.9.gz</span></code>. Use <code class="docutils literal notranslate"><span class="pre">zless</span></code> or
<code class="docutils literal notranslate"><span class="pre">zgrep</span></code> for older log files.</p>
<ul class="simple">
<li><p><strong>apache</strong> - <code class="docutils literal notranslate"><span class="pre">/var/log/apache2/lava-server.log</span></code></p></li>
<li><p><strong>django</strong> - by default <code class="docutils literal notranslate"><span class="pre">/var/log/lava-server/django.log</span></code> contains
errors and warnings from django.</p></li>
<li><p><strong>scheduler</strong> - <code class="docutils literal notranslate"><span class="pre">/var/log/lava-server/lava-scheduler.log</span></code></p></li>
<li><p><strong>worker</strong> - <code class="docutils literal notranslate"><span class="pre">/var/log/lava-dispatcher/lava-worker.log</span></code>.</p></li>
<li><p><strong>test jobs</strong> - <code class="docutils literal notranslate"><span class="pre">/var/lib/lava-server/default/media/job-output/</span></code>
individual files are in a directory named after the start time of the
corresponding job <code class="docutils literal notranslate"><span class="pre">&lt;year&gt;/&lt;month&gt;/&lt;day/$ID</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">2018/08/10/1234</span></code>.
The directory includes the validation output <code class="docutils literal notranslate"><span class="pre">description.yaml</span></code> and the
full log file <code class="docutils literal notranslate"><span class="pre">output.yaml</span></code>.</p></li>
</ul>
</section>
<section id="command-line-debugging">
<span id="debugging-cli"></span><h3>Command line debugging<a class="headerlink" href="#command-line-debugging" title="Permalink to this heading">¶</a></h3>
<ul>
<li><p><strong>lava-server</strong> - <code class="docutils literal notranslate"><span class="pre">sudo</span> <span class="pre">lava-server</span> <span class="pre">manage</span> <span class="pre">shell</span></code>.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="development.html#developer-access-to-django-shell"><span class="std std-ref">Developer access to django shell</span></a></p>
</div>
</li>
<li><p><strong>lava-dispatcher</strong> - The actions of <code class="docutils literal notranslate"><span class="pre">lava-worker</span></code> can be replicated
on the command line. The relevant device configuration can be obtained using
<code class="docutils literal notranslate"><span class="pre">lavacli</span></code>, e.g.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ lavacli devices dict get --render DEVICE_HOSTNAME
</pre></div>
</div>
<p>This config can then be passed to <code class="docutils literal notranslate"><span class="pre">lava-run</span></code>, in this example in a
file named <code class="docutils literal notranslate"><span class="pre">device.yaml</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo lava-run --device device.yaml --output-dir /tmp/debug/ job.yaml
</pre></div>
</div>
<p>Every job is validated before starting and the validate check can be run
directly by adding the <code class="docutils literal notranslate"><span class="pre">--validate</span></code> option:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo lava-run --job-id 0 --device device.yaml --validate --output-dir /tmp/debug/ job.yaml
</pre></div>
</div>
<p>The job will not start when <code class="docutils literal notranslate"><span class="pre">--validate</span></code> is used - if validation passes,
the complete pipeline will be described. If errors are found, these will be
output.</p>
</li>
</ul>
</section>
<section id="configuration-files">
<span id="debugging-configuration"></span><h3>Configuration files<a class="headerlink" href="#configuration-files" title="Permalink to this heading">¶</a></h3>
<ul>
<li><p><strong>lava-server</strong> - <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code> - restart <code class="docutils literal notranslate"><span class="pre">apache</span></code>
and <code class="docutils literal notranslate"><span class="pre">lava-server</span></code> if this is changed. Holds details for django settings
including the authentication methods and site customization settings.</p></li>
<li><p><strong>jinja2 templates</strong> - <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/dispatcher-config/device-types</span></code>
These files are updated from <code class="docutils literal notranslate"><span class="pre">lava_scheduler_app/tests/device-types</span></code>
in the codebase. The syntax is YAML with jinja2 markup. Restart the
<code class="docutils literal notranslate"><span class="pre">lava-master</span></code> after changing the templates.</p>
<ul>
<li><p>to validate changes to the templates, use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ /usr/share/lava-server/validate_devices.py --instance localhost
</pre></div>
</div>
</li>
<li><p>to validate the combination of the template with the device
dictionary content, use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ lavacli devices dict get --render DEVICE_HOSTNAME
</pre></div>
</div>
</li>
</ul>
</li>
<li><p><strong>device dictionaries</strong> - <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/dispatcher-config/devices</span></code>
These files are specific to each instance and need to be named according to
the <code class="docutils literal notranslate"><span class="pre">hostname</span></code>.jinja2 of the device as configured on the same instance.</p></li>
</ul>
</section>
<section id="check-lava-instance">
<span id="check-instance"></span><h3>Check LAVA instance<a class="headerlink" href="#check-lava-instance" title="Permalink to this heading">¶</a></h3>
<p>Once a LAVA instance is installed admins can check for errors and warnings on the
deployed instance using:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo lava-server manage check --deploy
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">check</span> <span class="pre">--deploy</span></code> command uses the Django system check framework which is a
set of static checks to detect common problems and provide hints for how to fix
them.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/checks/">https://docs.djangoproject.com/en/3.2/ref/checks/</a> to know more
about Django system check framework.</p>
</div>
<p>If the JSON syntax of <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code> is invalid,
this check will display a warning: <code class="docutils literal notranslate"><span class="pre">settings.conf</span> <span class="pre">is</span> <span class="pre">not</span> <span class="pre">a</span> <span class="pre">valid</span> <span class="pre">json</span>
<span class="pre">file</span></code>.</p>
<p>LAVA sets the following values by default:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">SECURE_CONTENT_TYPE_NOSNIFF</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">SECURE_BROWSER_XSS_FILTER</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">SESSION_COOKIE_SECURE</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">CSRF_COOKIE_SECURE</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">CSRF_COOKIE_HTTPONLY</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">X_FRAME_OPTIONS</span> <span class="o">=</span> <span class="s1">&#39;DENY&#39;</span>
</pre></div>
</div>
<p>These values can be overridden in <code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code></p>
<p>The following checks are silenced and does not show any errors or warnings:</p>
<ul class="simple">
<li><p>SECURE_HSTS_SECONDS i.e., security.W004 which is documented in <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/settings/#secure-hsts-seconds">https://docs.djangoproject.com/en/3.2/ref/settings/#secure-hsts-seconds</a></p></li>
<li><p>SECURE_SSL_REDIRECT i.e., ‘security.W008’ which is documented in <a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/settings/#secure-ssl-redirect">https://docs.djangoproject.com/en/3.2/ref/settings/#secure-ssl-redirect</a></p></li>
</ul>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Admins should consult the respective Django documentation before changing these
values to suit the requirements of each LAVA instance.</p>
</div>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="installing_on_debian.html#django-localhost"><span class="std std-ref">Using localhost or non HTTPS instance URL</span></a></p>
</div>
</section>
<section id="displaying-current-settings">
<span id="diff-settings"></span><h3>Displaying current settings<a class="headerlink" href="#displaying-current-settings" title="Permalink to this heading">¶</a></h3>
<p>The django developer shell can be used to check the value which is
currently set:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ sudo lava-server manage shell
Python 3.6.7 (default, Oct 21 2018, 08:08:16)
[GCC 8.2.0] on linux
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
(InteractiveConsole)
</pre></div>
</div>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">settings</span><span class="o">.</span><span class="n">CSRF_COOKIE_HTTPONLY</span>
<span class="go">False</span>
</pre></div>
</div>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="development.html#developer-access-to-django-shell"><span class="std std-ref">Developer access to django shell</span></a></p>
</div>
<div class="admonition caution">
<p class="admonition-title">Caution</p>
<p>Avoid making changes to the defaults inside the LAVA code.
These will be replaced when you upgrade to future versions of the
lava-server package. If you need to make changes, instead edit
<code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code> where they will be preserved.
Whenever you make changes, be sure to restart the LAVA daemons
before checking for the effects of the change:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo service lava-server-gunicorn restart
</pre></div>
</div>
<p>In some situations, you may also need to restart <code class="docutils literal notranslate"><span class="pre">lava-scheduler</span></code>,
<code class="docutils literal notranslate"><span class="pre">lava-server-gunucorn</span></code> and <code class="docutils literal notranslate"><span class="pre">lava-publisher</span></code> in the same way.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p>Installing a new release or a set of <a class="reference internal" href="debian.html#testing-packaging"><span class="std std-ref">Developer
packages</span></a> will also restart all LAVA daemons.</p>
</div>
</div>
<section id="django-support-in-1-11">
<h4>Django support in 1.11<a class="headerlink" href="#django-support-in-1-11" title="Permalink to this heading">¶</a></h4>
<p>If your local instance is using <code class="docutils literal notranslate"><span class="pre">python3-django</span></code> from <code class="docutils literal notranslate"><span class="pre">buster</span></code>, the new
support in 1.11 and later can be used to display the settings as well:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo lava-server manage diffsettings --all
</pre></div>
</div>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/django-admin/#diffsettings">https://docs.djangoproject.com/en/3.2/ref/django-admin/#diffsettings</a></p>
</div>
<p>Settings that don’t appear in the defaults are followed by “###”,
e.g.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">BRANDING_MESSAGE</span> <span class="o">=</span> <span class="s1">&#39;Master branch instance on Debian Buster.&#39;</span>  <span class="c1">###</span>
</pre></div>
</div>
<div class="admonition danger">
<p class="admonition-title">Danger</p>
<p>Access to <code class="docutils literal notranslate"><span class="pre">lava-server</span> <span class="pre">manage</span></code> needs to be restricted
in the same way as read permissions on
<code class="docutils literal notranslate"><span class="pre">/etc/lava-server/settings.conf</span></code> as the current settings will
include the <strong>database secret_key</strong>, authentication passwords for
services like LDAP and other sensitive information.</p>
</div>
</section>
</section>
</section>
</section>


    </div>
      
  </div>
</div>
<footer class="footer">
  <div class="container">
    <p class="pull-right">
      <a href="#">Back to top</a>
      
    </p>
    <p>
        &copy; Copyright 2010-2019, Linaro Limited.<br/>
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br/>
    </p>
  </div>
</footer>
  </body>
</html>